Skip to content

Support R 4.x, modernise the build, and bring the console up to current libraries - #220

Open
SamLovick wants to merge 85 commits into
sdllc:masterfrom
SamLovick:console-upgrade
Open

Support R 4.x, modernise the build, and bring the console up to current libraries#220
SamLovick wants to merge 85 commits into
sdllc:masterfrom
SamLovick:console-upgrade

Conversation

@SamLovick

@SamLovick SamLovick commented Sep 3, 2026

Copy link
Copy Markdown

Summary

Makes BERT work with current R (tested with 4.5.2) without dropping older R, moves the native build onto a current toolchain, and brings the console up to current versions of Electron, TypeScript, Monaco, xterm, rxjs and protobuf. Addresses #194, #207 and #124.

R controller

  • The controller refused any R other than 3.5. It now checks the version at startup: a floor of 3.5, a warning above the tested major series, no ceiling. One ControlR.exe, built against R 4.5.2, starts under R 3.5.0, 4.2.2 and 4.5.2.
  • String encoding is explicit on every R version: strings from Excel and the console are marked UTF-8 when handed to R, strings taken from R are translated to UTF-8, and console text is converted to and from the Windows code page only when R's native encoding is not UTF-8.
  • ControlR.exe carries the UTF-8 code-page manifest that R's own executables use. Without it, R 4.2+ embedded in another process runs on the system code page, and strings outside it cannot exist in the session.
  • Console messages are built from the length R supplies rather than a terminating NUL; the old code could send stale bytes after the text.
  • R_LEGACY_RCOMPLEX is defined because MSVC cannot compile the _Complex member R 4.3 added to Rcomplex.

Build

  • v145 toolset, Windows SDK from the installed VS, ARM64EC configurations alongside x64.
  • protobuf 5.29 via vcpkg in manifest mode; generated code regenerated, .proto unchanged so the wire format is unaffected.
  • The R installation is an overridable MSBuild property (BertRHome); import libraries regenerated from R 4.5.2 with their .def files checked in, and R's exported variables marked DATA so the ARM64X import libraries link.
  • controlr.cc is stored as UTF-8 (it was UTF-16, which git shows as binary).

Console

  • Electron 1.8 → 44, TypeScript 2.7 → 7, Monaco 0.10 → 0.56, xterm 3 → 6, rxjs 5 → 7, google-protobuf 3 → 4, and the smaller libraries. The architecture is unchanged (Node in the renderer as an explicit opt-in, @electron/remote for the few main-process calls; navigator.clipboard replaces the removed renderer clipboard module).
  • The xterm port no longer patches xterm's prototype; the two remaining reaches into its internals (cell size, scrollback trim event) are isolated in one file.
  • The release update check is skipped for tagged development builds, and the add-in's version string carries a branch tag and build time so a local build is distinguishable in the console.
  • npm with a committed lockfile instead of yarn.

Docs

docs/BUILDING.md describes the build, the R version policy and what has been exercised; docs/MODERNISATION.md records what changed in R and the toolchain since 2018 and the remaining work (installer still bundles R 3.5, Julia 0.x controllers, strict TypeScript, Monaco as ES modules, file paths from the add-in in the code page).

Verification

  • Full solution builds Release x64 (ControlR.exe, BERT64.xll, BERTRibbon2x64.dll); ControlR also builds Release ARM64EC.
  • Used with Excel and R 4.5.2: functions from cells, the console shell and editor, copy/paste, and non-ASCII strings ("café ñ 日本" round-trips through both a cell and the shell; nchar("café") is 4).
  • R 4.2.2 and 3.5.0 have had the controller's startup check only.

Generated with the assistance of Claude Code

Sam Lovick and others added 18 commits September 3, 2026 11:17
The R 4.x work introduced several large, fully regenerable trees that
should never enter version control:

  - vcpkg_installed/ under BERT, ControlR, ControlR-45 and Ribbon
    (~750 MB each, rebuilt from vcpkg.json)
  - ARM64EC/ and ARM64/ build output alongside the existing x64/Win32
  - R-3.5.0/ and R-4.5.2/, which are local R installations used only as
    build inputs (headers and import libraries)
  - .vs/ per-machine state and .vscode/ editor-local settings

Also ignores package-lock.json under Console, which uses yarn; carrying
both lockfiles produces divergent dependency installs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tree previously built against a source checkout of protobuf 3.5.0
sitting next to the repository, with each project hard-coding a relative
path into ../../protobuf-3.5.0. That version predates R 4.x and no longer
compiles cleanly with a current toolset.

Protobuf is now consumed through vcpkg in manifest mode. Each native
project gains a vcpkg.json declaring the dependency, and a shared
vcpkg-configuration.json pins the registry baseline so the resolved
version is reproducible.

Regenerated the protobuf output to match:

  - PB/variable.pb.cc and PB/variable.pb.h with protoc 5.29.5
  - Console/generated/variable_pb.js with the 3.21 JavaScript generator,
    and bumped the google-protobuf dependency to ^3.21.4

The .proto schema itself is unchanged, so the wire format between the
add-in, the console and the language controllers is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file was UTF-16, which git treats as binary, so no change to it
could be reviewed as a diff. The content is plain ASCII; only the
encoding changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Move the projects from v141 to v145 and from a pinned Windows SDK to
the latest installed one, enable vcpkg manifest mode so protobuf comes
from the manifests added in the previous commit, and add Debug and
Release ARM64EC configurations alongside x64. The solution header moves
to Visual Studio 18 accordingly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The controller refused to start unless the hosted R reported exactly
3.5, which is why BERT could not be pointed at a newer R even though
the embedding interface it uses had not changed. Reject only versions
older than the floor this code is built against, and warn rather than
refuse on a major series newer than the one the build was tested with.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
R for Windows switched its native encoding to UTF-8 in 4.2.0. Before
that, console output arrived in the Windows code page and was converted
to UTF-8 on the way to the console, after a validity check that let
already-UTF-8 output through. With a UTF-8 R that check is at best
redundant and at worst harmful, because a multi-byte character split
across two writes fails it and gets mangled by the conversion. Decide
once at startup, from the version R reports, whether output is passed
straight through, so one binary behaves correctly on both sides of the
4.2.0 change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Take the R headers and import libraries from a single, overridable
location: BertRHome, defaulting to R-4.5.2 beside the projects and
settable with /p:BertRHome=<path>. Replace the checked-in R 3.5 import
libraries with ones generated from R 4.5.2, and check in the .def files
they were built from so the export lists are reviewable. Add ARM64X
import libraries for the ARM64EC configurations.

Define R_LEGACY_RCOMPLEX in every configuration: from 4.3.0,
R_ext/Complex.h defines Rcomplex as a union with a C99 _Complex member,
which MSVC does not support, and the legacy definition is the
documented escape hatch. The layout is identical, and this code only
reads and writes the r and i fields.

RebuildLibs.ps1 learns to build the ARM64X libraries, gains a usage
note, and loses a stray debugging line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two fixes in the console, both surfaced by running against a current CRAN
and a current R.

The package chooser scrapes available_packages_by_name.html and took the
anchor text as the package name. CRAN now wraps some names in
<span class="CRAN">, so those entries appeared in the list with raw
markup around them. Names are stripped of any tags rather than that one
span, and empty results are skipped, so a further change to the page
layout degrades instead of producing garbage entries.

MessageUtilities.VariableToObject logged every protobuf value case it
could not translate. R 4.x sends several of these, and the result was a
devtools console too noisy to read. The log is now behind a BERT_DEBUG
environment variable, following the BERT_VERSION convention already used
in update_check.ts, so the diagnostic survives but is off by default.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… roadmap

docs/BUILDING.md covers the toolchain, how the R installation is chosen
(BertRHome), the import libraries and how to regenerate them, running
against an R other than the bundled one, and which R versions the
controller accepts and which have actually been exercised.
docs/MODERNISATION.md records what changed in R and the toolchain since
2018, what this branch does about each change, and the remaining work in
a suggested order. The README points at both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ries link

lib treats every export in a .def as a function unless told otherwise.
x64 gets away with that because R's headers import variables through
__declspec(dllimport), but the ARM64X libraries need to know which
exports have no code behind them, or the ARM64EC link fails with
LNK1373 looking for an exit thunk for R_DimNamesSymbol. RebuildLibs.ps1
now marks every name R's headers declare with LibExtern, plus the
GA_-prefixed spellings graphapp.h maps them onto, and the .def and .lib
files are regenerated from R 4.5.2. The x64 build is unaffected; the
ARM64EC configuration now links.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The console's Help menu shows the version string the add-in exports, and
until now a locally built add-in was indistinguishable from the 2.4.3
release. Split the define into the release number, which the installer
script reads, and the string the console shows, which now carries a
branch tag and the compile time of bert.cc.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The console asks bert-toolkit.com for the latest release and offers it
whenever the add-in's version is lower. A build from a development
branch carries a tag after its release number and is not a release, so
the comparison is meaningless; on the r4-support branch it offered the
2018 release as an upgrade. Return before fetching when the version is
not a plain number.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The console was pinned to Electron 1.8, TypeScript 2.7 and Monaco 0.10.
This brings the platform up to date while keeping the console's
architecture: the renderer still talks to the add-in over named pipes
and reads and watches files itself, which current Electron allows as an
explicit choice (nodeIntegration on, contextIsolation off), and the
removed remote module is replaced by @electron/remote.

Changes forced by the new versions:

  - dialogs return promises now; the editor and shell use the *Sync
    variants, which keep the existing control flow
  - clipboard.readText is asynchronous; Paste awaits it
  - Menu.popup takes an options object
  - modules exported as a function (markdown-it, less) are imported as
    defaults, since the namespace shim only copies own properties
  - Promise sites whose resolve carries no value are typed Promise<void>
  - Monaco's JSON defaults moved to a top-level namespace, and the
    model's language comes from getLanguageId() instead of a private
    field
  - the AMD loader is loaded after the console's own code, so that
    UMD-packaged libraries register with node rather than with the
    loader's global define, and Monaco loading waits for the loader
    script if it gets there first

Also: the "first run" path no longer rejects when an example file is
missing, the version parser ignores the tag a development build
carries, strict mode is explicitly off (the code predates it; making it
strict-clean is separate work), and the package manager is npm with a
committed lockfile instead of yarn.

Verified by launching the console from source: the welcome page
renders, a file opens through the File menu and Monaco highlights it.
The terminal needs Excel and is not covered yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rxjs 5 to 7: operators move into pipe(), Observable.create becomes the
constructor, and merge is a top-level function. Every site changes the
same way.

xterm 3 to 6: the terminal no longer patches xterm's prototype or
imports its internals. The fit, cursor-position and annotation addons
become plain functions and a class that use the public buffer API
(cursorX, cursorY, viewportY, baseY, getLine), the fit addon for the
proposed geometry, and the web-links addon. Cell sizes and the
scrollback trim event have no public equivalent, so xterm_internals.ts
holds those two reaches into xterm's core, with a fallback, and nothing
else touches it. Horizontal scrolling of wide output, which used to
shift xterm's screen element under its own viewport, now scrolls the
terminal's container instead; the fit still keeps the column count at
least as wide as the widest line. xterm stopped exporting its character
width table, so wcwidth comes from the package of that name.

google-protobuf 3.21 to 4.0 (the generated code is unchanged), chokidar
2 to 4 (5 is ESM-only), markdown-it 8 to 15, less 3 to 4, js-base64 2
to 3, all without source changes beyond the imports already made.

Verified by launching from source: startup is clean and the editor
works. The terminal needs Excel and is covered next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he version

Electron no longer provides its clipboard module to renderer processes,
so Copy in the shell, Paste, and the editor's copy-path command threw
on an undefined module. The standard navigator.clipboard API is what
Electron recommends instead; a small wrapper keeps the call sites as
they were. Checked in a bare renderer: the round trip works, and xterm
keeps its selection across the right-click that opens the context menu.

startup.R split BERT_VERSION on dots and coerced every part to a
number, so the tag a development build now carries produced an NA
warning that R reported at the next prompt. Parse the leading numeric
part only; the full string is still kept as version.string.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Strings from the add-in and the console are UTF-8 whatever R is hosted:
the add-in converts Excel's UTF-16 strings and COM names to UTF-8, and
the console is a web page. The controller used to hand them to R as
native strings, which is right from 4.2.0 and wrong before, and took
strings back from R with CHAR(), guessing at the encoding with a
validity check and a code-page conversion.

Now strings into R are marked UTF-8 (Rf_mkCharLenCE with CE_UTF8) and
strings out of R go through Rf_translateCharUTF8, which honours the
encoding mark each string carries. Console keyboard input is converted
to the Windows code page under an R older than 4.2.0, the mirror of the
existing output conversion, keyed off the same startup check. The one
string left native is the path given to source(): the add-in produces
it with the ANSI file APIs, and the fix for that belongs on its side.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two things stood between the encoding work and a working shell.

R 4.2.0 and later use UTF-8 as their native encoding only in a process
that has the UTF-8 code page, which R's own executables get from a
manifest. Embedded in ControlR.exe, R was running on code page 1252, so
the version check alone was the wrong test for UTF-8 pass-through:
nchar("café") gave 5. ControlR.exe now carries the same manifest
setting, and the pass-through is decided by the version and GetACP()
together, with the code-page conversion as the fallback.

ConsoleMessage built the protobuf string from R's buffer as if it were
NUL-terminated, ignoring the length R passes. Stale bytes after the
text went out with it; the old JavaScript protobuf runtime substituted
characters silently, the current one rejects the message, and the
prompt that followed was lost, leaving the shell busy for ever. The
console now also logs and skips a frame it cannot decode instead of
abandoning the rest of the batch.

The console gains a BERT_CONSOLE_DEBUG_PORT environment variable that
exposes the DevTools protocol, which is how this was diagnosed and
verified: l10n_info() reports UTF-8, nchar("café") is 4, and
"café ñ 日本" round-trips through the shell.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sam Lovick and others added 11 commits September 3, 2026 15:21
…se script

The NSIS script now installs over an existing BERT 2 without touching
bert-config.json, user-stylesheet.less or Documents\BERT2 (created from
templates only when absent, as before), replaces the application-owned
console, module and startup directories outright, and ships only what
the x64 build produces: no 32-bit add-in, no Julia controllers, no
bundled R. It finds the installed 64-bit R in the registry and writes it
into bert-languages.json as the last candidate home, after the R-3.5.0
an older installer may have left, so an upgraded installation moves to
the installed R without editing a file. BERT.R.home still overrides.

build-release.ps1 replaces the 2018 build-installer.ps1 (Visual Studio
2017 paths, code signing, 32-bit builds): it builds the native projects,
installs the BERTModule R package into Build\module with the registered
R, packages the console, runs makensis, and zips the same files together
with INSTALL-FROM-ZIP.md for people who copy by hand.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…butable

The add-in, ribbon and controller already use the static C runtime, but
protobuf and abseil came from vcpkg's x64-windows triplet as DLLs built
against the dynamic runtime. On a machine without the Visual C++
redistributable the add-in could not load its dependency and Excel
dropped it silently: the ribbon appeared, R functions gave #NAME?, and
the console would not open. The x64 configurations now use the
x64-windows-static triplet, so protobuf and abseil are linked in and
the shipped files import only Windows system DLLs.

The installer stops shipping the two DLLs and removes them from an
earlier installation; the release script refuses to package a binary
that still imports the runtime or protobuf DLLs, and finds makensis
where the NSIS installer puts it. Build tag bumped to -r5 so the
corrected release is distinguishable from -r4.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three separate faults, all in the editor pane:

The right-click menu was built by asking monaco's contextmenu
contribution for its entries. That is private API, and in monaco 0.56 the
method needs arguments it was not given, so the handler threw and no menu
appeared. The entries are listed explicitly now, with the execute
commands this class registers per language appended, so the menu no
longer depends on monaco's internals.

Monaco no longer registers cut, copy and paste actions at all. It only
ever registered them where the document.execCommand editing commands
were available, and chromium has withdrawn those from web content. Cut,
copy, paste and select all are therefore implemented here, against the
model and the clipboard.

Since monaco moved to the EditContext input surface, the browser's own
paste no longer reaches the editor: a plain textarea in the same window
receives Ctrl+V, the editor does not. The clipboard shortcuts are bound
in monaco, which also stops the browser acting on the same keystroke.

The Edit menu's cut, copy, paste, find and replace items were never
wired to anything; they fell through to a debug log. They now act on
whichever pane last had the focus, remembered as the menu itself takes
the focus away. The clipboard helper falls back to the main process
clipboard, because the web clipboard API refuses to run while the
document is not focused, which can happen around native menus.

Verified over the devtools protocol: menu and keyboard cut, copy and
paste, save writing to disk and clearing the dirty flag, and the context
menu handler running without error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…or IntelliSense

A function described with a single string was silently ignored. The
controller only read the description attribute when it arrived as an
array, and a character vector of length one crosses the wire as a scalar,
so attr(f, "description") <- "text" did nothing while c("text", "") worked.

docs/FUNCTION-HELP.md now covers the whole facility: how to write the
descriptions, where Excel shows them (the Insert Function and Function
Arguments dialogs), and why the inline tooltip is not among them. Excel
draws that tooltip only for its own functions, and gives an add-in no way
to produce it. The example functions and the console welcome page point
at it, since none of this was written down anywhere.

For the tooltip, the add-in now writes its registered functions and their
descriptions to BERT-IntelliSense.intellisense.xml, in the format the
Excel-DNA IntelliSense add-in reads. That add-in draws the tooltip Excel
withholds. It looks for the file beside a loaded workbook or workbook
add-in and ignores XLL add-ins, so BERT-IntelliSense.xlam is shipped as
the thing to find it beside: an empty add-in with no code. Both are
optional, and BERT does not depend on either.

Verified with Excel and R 4.5.2: a function described with one string and
one described argument by argument both reach the generated file, with
the reserved characters escaped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Loaded the carrier add-in and Excel-DNA IntelliSense 1.9.0 into 64-bit
Excel with BERT: typing =R.TestAdd( shows the signature with the current
argument in bold, the function description and the argument description,
taken from the description attribute on the R function. Notes the
version tested and that the add-in is not code-signed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…for R

The components page now offers "Function help in the formula bar", off by
default. Selecting it installs the Excel-DNA IntelliSense add-in and the
carrier add-in, and adds both to Excel's add-in list; running the installer
again with the box cleared takes them out again, and uninstalling BERT
removes them too. /HELP-FEATURE and /NO-HELP-FEATURE set it from the
command line, for scripted installs.

Excel keeps its add-in list as OPEN, OPEN1, OPEN2 and so on, with no gaps.
The installer adds to the first free slot, skips entries already there, and
on removal moves the last entry into the gap, so add-ins the user already
had are left alone. The IntelliSense add-in is MIT licensed and ships with
its licence; the build script downloads it, pinned by version and checked
against its hash, so the binary is not in the repository.

R detection was one registry value: the InstallPath the current R records.
An R that did not write it, or wrote it and was then removed, looked like
no R at all, and the installer said so even where R was installed. It now
looks at that value for both R64 and R, per user and per machine, then at
the per-version subkeys below them, then in Program Files, and keeps the
newest it finds. It says nothing unless there is no R at all, or the one it
found is older than 4.2, and it says which one that is.

Message boxes gained silent defaults, so an unattended install cannot sit
waiting behind a dialog.

Verified on this machine: installing with the feature registers both
add-ins and the tooltip works in the session straight afterwards; installing
without it removes them and leaves the three add-ins already registered
untouched; three installs in a row produce no duplicates; and a per-user R
registered only as a version subkey is found and preferred when it is newer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closing the console does not close it: the window cancels the close and
asks the add-in to hide it, so the ribbon command normally just shows the
window again. That path hid the window, rewrote its window styles by hand
(moving it between an application window and a tool window, and setting
the visible bit itself), and showed it again. Rewriting the styles leaves
the window out of step with what chromium believes about it, and under
current chromium the console came back with a blank client area that took
no input: a title bar and menus over nothing. It now shows and hides the
window and leaves the styles alone, which is all it needed to do; a
hidden window has no taskbar button anyway, which is what the style
changes were for.

The process id was also never cleared, so if the console had gone away
the ribbon command would look for a window that no longer existed and do
nothing at all, for the rest of the Excel session. It now checks whether
the process is still running and starts a new console if it is not.

Reproduced and fixed against Excel: open the console, close it, open it
again, and it comes back with the editor and shell as it was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Install Packages sat on "loading package list" for ever. The list URL was
built with path.join, which is for file paths: on windows it turned the
mirror into ".\https:\cloud.r-project.org\web\packages\..." and the fetch
failed. It is a URL, so it is joined as one now, and a fetch that fails
says so in the dialog instead of leaving it waiting.

getOption("repos") also does not always arrive in the same shape. R holds
it in a named character vector, which reaches the console as an object,
but a vector of one loses its names on the way and arrives as a plain
string, which is what happens when CRAN is the only repo set. The code
only handled the object, so it either offered the mirror list again or
threw. It now accepts either. Choosing a mirror no longer writes the
string "undefined" into the option as CRANextra when there was none.

The list is now two columns, a fixed one for the package name and the
rest for the description, so the descriptions line up down the list;
either can be clipped, and the full text is in the tooltip.

One notch of the mouse wheel used to scroll the list all the way to the
end. The list recycles its rows as it scrolls, so the content above the
viewport keeps changing size; chromium's scroll anchoring corrects the
scroll position to compensate, which fires another scroll event, which
recycles more rows. Scroll anchoring is switched off for these lists.
Measured: one notch now moves 300px and stops, where it used to climb
about 6000px every 400ms until it ran out of list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The terminal already puts itself on the window object, and the editor was
added alongside it. The menu utilities go there too, so the menu commands
can be driven and inspected while testing the interface. All three are
behind the dev flags the add-in passes, so a normal install exposes
nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closing a modified file discarded the changes without a word; the code has
carried a "FIXME: warn if dirty" since 2018. Closing one now asks whether to
save, discard or keep it open, from the File menu and from the tab's own
close button alike. An unmodified file still closes straight away.

Saving from that question has to be able to fail -- an empty document opens
the save-as dialog, which the user can cancel -- so SaveTab now returns a
promise that resolves true only once the file is on disk, and the tab stays
open if it does not.

Close All and Close Others ask one file at a time and stop at the first file
the user decides to keep, rather than firing every question at once.

Alerts had a related fault: the outgoing alert hides itself when its fade-out
transition ends, which would hide an alert opened in the meantime, leaving a
modal nobody could answer. The next alert now calls off the pending fade.

Verified against the packaged console over the devtools protocol: cancel,
discard and save from the menu and from the tab button, escape, an
unmodified file, and Close All over five modified files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With three answers -- save, don't save, cancel -- the labels ran together
into what read as a single line of words on one green block.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sam Lovick and others added 25 commits September 4, 2026 19:56
r10 shipped modules for 4.5 and 4.6, so drawing failed on 4.2 through 4.4 and
3.5 lost Excel references as well. Modules now ship for 3.5, 4.2, 4.3, 4.4,
4.5 and 4.6 -- the whole range the controller runs on -- and they come to
about 3 MB together.

What made this practical was building them on runners rather than here: each
series needs its own R and its matching Rtools, which is a gigabyte-plus of
downloads per series on a build machine. build-release.ps1 -FetchModules
takes them from the last successful run of the "graphics modules" workflow,
checking each artifact really was built for the R series it claims before it
goes into the package -- the workflow checks the same thing, because a module
relinked from stale objects will otherwise claim one series and behave as
another.

The gh calls name the repository explicitly: this clone has an upstream
remote too, and gh resolves there by default, where the workflow does not
exist.

Verified from the installer in Excel on 3.5.0, 4.2.2, 4.5.2 and 4.6.1:
drawing and Excel references work on all four. 4.3 and 4.4 ship modules built
and checked on CI but are not tested here for want of those R versions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Excel has no complex type, so BERT sends one as text in the a+bi form Excel's
own engineering functions read. The conversion used a default stringstream,
which formats to six significant figures, so

  314159.2653589793 + 2.718281828459045i

reached the cell as "314159+2.71828i" -- and IMREAL of it returned 314159.
No error, no warning; the digits were simply gone. Doubles were never
affected, since they cross as binary doubles.

Set 15 significant digits, which is what Excel keeps internally and what R
prints. Checked in Excel: the value now shows all 15 digits, Excel's IM*
functions still parse it, and IMREAL of a complex 1/3 returned from R minus
1/3 computed in the sheet is exactly zero.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In rendered markdown -- the release notes and the welcome page -- paragraphs
are padded 10px on the left while h2 and h3 were padded 16px, so every
subheading sat further in than the text it introduced. All three heading
levels now share the paragraph indent.

Measured after the change: h1, h2, h3, p, table and pre all start at the same
left edge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things in rendered markdown, following on from the heading indent.

A table carries no left padding of its own, so while its box lined up with
the paragraphs, its first column started ten pixels further left than the
text above it and read as an outdent. It now has the same left offset, with
cell padding and a rule under the header row so the columns are legible.

List items ran together. They now get 1em beneath them, which measures as the
same 14px gap that separates paragraphs.

Measured after the change: heading, paragraph and table text all start at the
same left edge, and the gap between list items matches the gap between
paragraphs exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
So they are not lost between now and whenever r13 is cut.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Text started hard against the edge of the pane. Inset the scrolling
container by 6px rather than the terminal element inside it: the fit
addon sizes columns from the scrolling node's width, so a narrower
terminal would overflow it and add a spurious horizontal scrollbar.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Build

Build\bert2.ico was an untracked duplicate of the checked-in Install copy,
and the release build fails without it. The build now copies it, and Build
ignores the copy, so a clean tree stays clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Excel gives a function one line in its dialogs and no way to show more.
It does have a help topic -- the "Help on this function" button -- but it
takes a URL or a .chm and nothing else, and BERT was filling that slot
with the registration index, which is not a topic and did nothing.

So: startup.R reads the `#'` block above each definition (a comment is
not part of the closure, so the file has to be read after it is sourced),
renders it into a self-contained page under <home>\help, and the add-in
serves those pages from a loopback listener and registers the address as
the topic. The IntelliSense XML gets the same address as HelpTopic, which
that add-in draws as a link in the tooltip.

The block also fills in the one-line description and the argument
descriptions when the function has no description attribute, so a
function is documented once and both routes agree. attr(f, "help") takes
the same text, and attr(f, "help.url") points somewhere else entirely.

Two fields on the function descriptor carry it: help_url and help_file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Open files are remembered across sessions, but the text came back from
localStorage. If the file changed while the console was not running -- an
edit elsewhere, or a different file saved over it -- the tab showed the
old contents with no sign of it, and a deleted file came back as though
it still existed.

Restored tabs now read from disk, and a tab whose file has gone is not
restored at all (its cache entry goes too). A modified, never-saved tab
keeps its text: that only exists in the cache, so throwing it away would
lose work. Since a dropped tab counts as removed, the unclose list is
cleared after the restore, so a deleted file cannot be unclosed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Open and Save As started wherever the shell last left them, which is
rarely where the R files BERT loads live. Both now start at
functionsDirectory from bert-config.json, expanding environment variables
the way the add-in does when it reads the same setting. Save As only
suggests it for a document that has never been saved; one with a path
still opens where it lives.

Falls back to the previous behaviour when the setting is absent or names
a directory that no longer exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Excel fixes arity at registration and calls a distinct entry point per
function, so the pool of dispatchers is compiled with a fixed parameter
count -- 16 since the beginning, with the seventeenth argument silently
not passed. The count is now BERT_MAX_ARGUMENTS, one constant, with the
type string and the xlfRegister parameter block derived from it, and the
parameter list written once as a macro list so the four places that
repeat it cannot drift. BERT.Call gets the same width.

64 rather than Excel's ceiling of 254 because every one of the 2048
dispatchers carries the full width: 16 -> 64 costs 1.4 MB of code.

Two things the wider functions exposed, both fixed here: an argument-name
list longer than 255 characters makes excel reject the registration
outright, so it is now cut at the last name that fits; and building that
list for a function with no arguments read past the end of an empty
string.

Verified in Excel with functions of 16, 17, 20, 40, 64 and 70 arguments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ribbon is a COM add-in: registered separately, disable-able by Excel,
and the only part of the install that touches anything outside BERT's own
directory. Dropping it turned out to need two changes, neither obvious
from the outside.

The ribbon LOADS the xll -- CConnect::OnConnection calls RegisterXLL, and
nothing else does, so with the ribbon disabled BERT does not load at all.
The installer now registers BERT64.xll in Excel's add-in list when the
ribbon component is cleared, and takes that entry back out when it isn't.

The ribbon was also the only source of Excel's Application pointer, which
drawing into a sheet and the EXCEL object in R both need; without it,
graphics silently drew nothing. The add-in now asks Excel for it directly
-- frame window, sheet pane, AccessibleObjectFromWindow with
OBJID_NATIVEOM -- the way an add-in with no COM component of its own has
to. It must happen at load: a first version acquired it lazily, from the
graphics callback, and asking for the object model mid-calculation hung
Excel outright.

Also: application_dispatch_ was never initialized, and it is now read to
decide whether we have a pointer yet.

Verified with the ribbon disabled: functions calculate, the console opens
(openConsole, now in the config template), the EXCEL object is there, and
a plot lands in the sheet -- one shape, exactly as with the ribbon.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Without the tab there is no console button, and the console hides rather
than closes when you click the X -- so there has to be a way back to it.
BERT.Console is a registered command and was registered with no shortcut;
it now takes CONTROL+SHIFT+R, which is the eighth field of the
registration table and was sitting empty.

Documented alongside the routes that already existed: openConsole at
startup, Application.Run from VBA, the name typed into the macro dialog,
and EXCEL$Application$Run from R. Also where the ribbon's own switch is
(COM Add-ins, or LoadBehavior in the registry) and the trap that turning
it off there leaves nothing loading the xll.

Verified with the ribbon disabled: console opens with Excel, the X hides
it and leaves the process running, BERT.Console shows it again. The
keyboard shortcut is not verified -- the harness cannot press keys.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With the ribbon installed both sources now fire: we take a pointer at
load, and the ribbon offers another as it connects. They are two pointers
to the same Application, and taking the second marshalled a second stream
over the first. First one wins instead.

The pointer we acquire ourselves comes back AddRef'd and nothing else
releases it, so track that and give it back in Close. A pointer from the
ribbon belongs to the ribbon and is left alone.

Verified on the default path -- ribbon enabled, xll loaded by it as
usual: a plot still lands in the sheet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…bbon

Clearing "BERT2 Ribbon Menu" in Excel's COM add-ins dialog switched the
whole add-in off, because the ribbon is what calls RegisterXLL and the
installer only registered the xll directly when the ribbon component was
cleared at install time. Turning it off afterwards -- the obvious thing to
try -- left no functions, no console, and nothing for Ctrl+Shift+R to
reach.

The xll now goes in Excel's own add-in list whatever the components page
said. The ribbon's RegisterXLL is then never reached, since it only runs
when the add-in is not already loaded, and turning the ribbon off in Excel
degrades to an xll-only BERT instead of no BERT.

Verified: installed with the ribbon component selected, then disabled the
COM add-in the way you would in Excel -- 65 functions registered and the
controller running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I documented ALT+F8 as a way to open the console, on the strength of a
line in the C API documentation about command names being usable
"anywhere a valid command name is required". It is not true of that
dialog: it resolves what you type in workbook scope -- Excel reports
"cannot run the macro '!BERT.Console'" -- and an XLL command belongs to
the add-in's context, not a workbook's.

The routes that do work: Application.Run and ExecuteExcel4Macro (the
latter is what the ribbon button uses), openConsole at startup, and the
registered shortcut. Added a VBA snippet for binding a key of your own,
which is the fallback if the registered shortcut turns out not to fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xlfRegister's shortcut field binds the key -- excel dispatches it -- but
then resolves the command in workbook scope, so pressing CONTROL+SHIFT+R
in r17 raised "cannot run the macro '!BERT.Console'" instead of opening
the console. An xll command is registered in the add-in's context; no
workbook has it.

ON.KEY at load instead, released in xlAutoClose. It sends the name
through the same path as ExecuteExcel4Macro, which is what the ribbon
button uses and has always worked. Sam confirmed by hand that the key now
opens the console; a keystroke is not something the harness can test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shortcut was confirmed by hand on the development machine, and r18
installs and runs on the second machine, which has no development tools.
Worth stating in the entry: a keystroke cannot be tested any other way
here, so that check is manual every time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@SamLovick

Copy link
Copy Markdown
Author

Another set of findings from this branch, again independent of the R 4.x work: the ribbon add-in and the XLL are more tangled than they look, and two of the three problems below affect anyone running BERT today, ribbon or no ribbon.

The motivation was an install with no COM component at all — one that registers nothing, has no entry in Excel's COM add-ins list for Excel to disable after a crash, and touches nothing outside its own directory. That turned out to need more than not shipping BERTRibbon2x64.dll.

The ribbon is what loads the XLL

CConnect::OnConnection calls Application.RegisterXLL, and nothing else in the tree does. BERT64.xll is never registered in Excel's own add-in list, so the COM add-in is the only thing that brings it in.

The visible consequence is on any installation: clear BERT2 Ribbon Menu in File ▸ Options ▸ Add-ins ▸ COM Add-ins, expecting to lose the tab, and you lose everything — no functions, no console, nothing. I hit this myself before I understood the mechanism, and the failure looks like a broken install rather than a deliberate switch.

The fix is on the installer side: register BERT64.xll in Excel's add-in list (the OPEN/OPEN1/… values under …\Excel\Options) on every install. The ribbon's own RegisterXLL is then never reached, since OnConnection only calls it when SetPointers fails to find the XLL already loaded.

The ribbon is the only source of Excel's Application pointer

BERT_SetPointers is exported by the XLL and called only by the ribbon, and application_dispatch_ is set nowhere else. It is used for two things:

  • BERTGraphics::CreateDeviceTarget / QuerySize, which draw into the sheet;
  • LanguageService::SetApplicationPointer, which is what puts the EXCEL object in front of R.

Both are guarded on the pointer being non-null, so without the ribbon they fail silently. Plotting from a cell completes without error and draws nothing at all.

An add-in with no COM component of its own can get that pointer directly: find this process's XLMAIN window (matched against xlGetHwnd), walk down to XLDESK then EXCEL7, and call AccessibleObjectFromWindow with OBJID_NATIVEOM, which returns the window's object model — Application hangs off it. Excel-DNA does the same thing.

One trap worth recording. It must happen while Excel is idle. My first version acquired the pointer lazily, on first use, and first use is a graphics update — which arrives while a cell is calculating. Asking for the object model at that moment hangs Excel outright, no error, no timeout. Doing it once in xlAutoOpen is fine. With the ribbon installed it never runs, because the ribbon has already handed a pointer over; where both fire, first one wins, or you marshal a second stream over the first.

xlfRegister's shortcut field does not work for XLL commands

Unrelated to the ribbon, but it came out of the same work. Registering BERT.Console with "R" in the shortcut field (the seventh of the registration strings) does bind CONTROL+SHIFT+R — Excel dispatches the key — but Excel then resolves the command in workbook scope and reports:

Cannot run the macro '!BERT.console'

An XLL command is registered in the add-in's context, so no workbook has it, and the name can never resolve that way. The same applies to typing the name into the macro dialog (ALT+F8), despite the C API documentation's line about command names being usable "anywhere a valid command name is required" — that is about XLM contexts, not that dialog.

ON.KEY works: Excel12(xlcOnKey, …, "^+r", "BERT.Console") at load, released in xlAutoClose. It routes the name the same way ExecuteExcel4Macro does, which is what the ribbon button has always used.

Where this is

All of it is on the fork, in the same branch as this PR, with the reasoning in docs/XLL-ONLY.md. The ribbon became an optional installer component; with it cleared you keep functions, the console, graphics, Excel references, the EXCEL object and function help, and lose the tab and the buttons BERT$AddUserButton adds, which are ribbon controls with nowhere else to live.

Tested both ways on Excel from Microsoft 365 with R 4.6.1: with the ribbon disabled, functions register, the console opens and comes back after being closed, and a plot lands in the sheet — one shape, the same as with the ribbon installed.

Happy to split any of these out as separate PRs if that is easier to review than the branch as a whole.

Sam Lovick and others added 4 commits September 10, 2026 19:39
That sentence was written for r10, which shipped two modules. r11 shipped
six -- 3.5, 4.2, 4.3, 4.4, 4.5, 4.6 -- and the changelog's support table
has said so since, but the README still told people drawing would not work
on their R. Also closed the gap where the next sentence ran into it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It sits beside CHANGELOG.md and Welcome.md now, in the installer and in
the zip. Taken from the repository root, not from Build: the README there
describes the build output directory and is not for shipping. The build
fails early if the root one is missing, like the rest of the payload.

Verified with a throwaway build: 6,459 bytes in the zip, and the same
file in the install directory after a silent install.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It ships with the install now, where relative links resolve to nothing:
the logo, the changelog, the build notes and the function-help page all
pointed at paths that only exist in a checkout. They are absolute URLs
into the fork, verified 200.

Added the ribbon section, since clearing that box is a choice the
installer now offers and the README said nothing about it: what you get
without the tab, what goes with it, and how to reach the console when the
button is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant